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Executive 

Summary 

Planning and conducting effective educational programs for adults 
entail contributions by various stakeholders including participants, 
instructors, coordinators, policy makers, and funders. They each 
make informal evaluative judgments. Sometimes reliance on such 
informal evaluation is insufficient because judgments may be based 
on inadequate information and are not shared with people who 
might use them to plan, improve, or justify the programs. 

This monograph reviews and synthesizes highlights from relevant 
writings on evaluation and suggests interpretations and applications 
for practitioners who plan and conduct various types of educational 
programs for adults. 

In this monograph, concepts, procedures, and examples from eval- 
uation reports have been selected to represent eight aspects: needs, 
context, goals, staffing, participation, programs, materials, and out- 
comes. Many evaluations focus on just one or two of these aspects. 
Conducting a program evaluation entails many decisions that can 
be grouped into eight broad action guidelines. The guidelines per- 
tain to purpose, stakeholders, planning, coordination, sources, data 
collection, analysis, and utilization. Many examples contain enough 
detail to portray the actual evaluation project and not just illustrate 
a guideline. 

Readers can use this overview to clarify basic evaluation concepts 
and procedures, locate publications likely to provide detailed assis- 
tance, and use suggested guidelines to conduct evaluations on se- 
lected program aspects of interest. Selecting a program on which to 
focus should entail selecting an issue of importance, being respon- 
sive to stakeholder interest, and considering available expertise and 
resources for evaluation. 

Information on program evaluation in adult and continuing educa- 
tion may be found in the ERIC database by using the following 
descriptor;*: * Adult Education, ^Continuing Education, Data Cob 
lection, Evaluation Criteria, ^Evaluation Methods, Needs Assess- 
ment, * Program Evaluation. Staff Development. Asterisks indicate 
particularly relevant descriptors. 
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Introduction 

Each educational program for adults is evaluated mfnrmally every 
time it occurs. This happens because the people associated with it 
make evaluative judgments. Adult participants do so when they 
assess the likely costs and benefits before they enroll, when they 
decide whether to persist or withdraw, and when they judge how 
beneficial the program was after completion. Instructors do so 
when they decide whether to teach a course or workshop, when 
they evaluate it in progress regarding possible modifications, and 
when they evaluate the costs and benefits to themselves when 
deciding whether to do so again. Coordinators do so when they 
receive feedback from instructors and participants and decide 
whether to offer the program again, modify it, or discontinue it. 
Funders and policy makers do so when they review information 
about program quality and outcomes and decide whether or not to 
continue to provide support. 

T here are several problems with reliance on such informal evalua- 
tion. Sometimes such evaluative judgments are based on inade- 
quate information. One purpose of more formal program evaluation 
is to arrange for more satisfactory evidence to undergird such deci- 
sions. A second problem is that such informal evaluative judgments 
are not usually shared with people who might use them to plan, 
improve, or justify the programs. Formal evaluations can use infor- 
mal assessments by various people who have a stake in a program, 
by validating, correcting, or supplementing them and by communi- 
cating conclusions. Thus, program evaluation entails various stake- 
bidders making evaluative judgments about various aspects of pro- 
gram quality and effectiveness, based on evidence, using procedures 
that encourage use of conclusions for program planning, improve- 
ment, and accountability. 

People in various roles are stakeholders regarding program plan- 
ning, implementation, and evaluation. Included are participants, 
instructors, coordinators, policy makers, and flinders. In many 
educational programs for adults, such stakeholders are unfamiliar 
with concepts and procedures for program evaluation. Sometimes 
the image of such evaluation is daunting because they think it has 
to be complicated, expensive, and risky. They seldom know that 
there are many evaluation concepts, procedures, and examples 
available that would enable them to strengthen their program 
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evaluation, with benefits that tar outweigh the costs ot the formal 
assessment. 

The purpose of this monograph is to review ami synthesize high- 
lights from relevant writings on evaluation and to suggest interpret 
tations and applications for practitioners who plan and conduct 
various types ot educational programs for adults. Most such practi- 
tioners are in roles as coordinators, instructors, directors, or policy 
makers. They are associated with all types ot providers of educa- 
tional programs for adults. Included are schools, community col- 
leges, universities, enterprises, labor unions, religious institutions, 
associations, penal institutions, and community agencies. As part of 
program development, practitioners can use evaluation for plan- 
ning, improvement, and accountability. Such evaluation activities 
could be on any level of effort and could focus on any aspect of the 
program. 

In practice and in writings about evaluation there has been sub- 
stantial attention to eight broad program aspects. In this mono- 
graph, concepts, procedures, and examples from evaluation reports 
have been selected to explore evaluation of these eight aspects: 
needs, context, goals, staffing, participation, programs, materials, 
and outcomes. Most evaluation projects focus on one or two of 
these aspects. To plan and conduct a specific program evaluation 
project entails many decisions and actions that can be grouped into 
eight broad action guidelines. The guidelines pertain to purpose, 
stakeholders, planning, coordination, sources, collection, analysis, 
and utilization. 

The heart of this monograph addresses applications. Its eight major 
parts are devoted to these action guidelines, and within each there 
are an illustrative rationale and various examples from evaluation 
reports drawn from some of the eight aspects on which an evalua- 
tion can focus. In each section, citations to pertinent writings from 
which the ideas were obtained allow readers to pursue in greater 
detail both the rationale and procedures. 

From books on evaluation and actual evaluation reports (both pub- 
lished and unpublished reports obtained from colleagues through- 
out the field) ideas and examples were selected that might help 
practitioners in various provider agencies strengthen their evalua- 
tion projects. 

Most evaluation projects focus on one or a few aspects or parts of 
an educational progrrm. In contrast, many writings on evaluation 
deal generally with the evaluation process and assume that the 
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reader will apply the generalizations to assessing the aspect of inter- 
est. The examples for each guideline explain and illustrate how 
general guidelines (which pertain to decisions such as ways to in- 
volve stakeholders or procedures for data collection) apply to plan- 
ning and conducting a specific evaluation project. Over the years, 
writings on evaluation have given increased attention to such spe- 
cific aspects. 

The combination of action guidelines and program aspects should 
enable a practitioner to focus on evaluation issues of practical in- 
terest in the local program setting. Various examples are included. 
They reflect not only action guidelines and program aspects, but 
also various types of educational programs for adults and the com- 
plexity of a project and resources required. In seeking pertinent 
examples, earlier reports were sometimes included along with more 
recent publications. In the published materials, the bibliographic 
citations allow reading in greater detail about concepts or proce- 
dures of interest. Also included are some unpublished examples 
that sometimes can illuminate the process better than reports that 
people are willing to publish. 



Theme and 
Variations 



In this chapter, two examples illustrate the main concepts and pro- 
cedures regarding evaluation of various types of educational pro- 
grams for adults. These two examples introduce eight guidelines 
that constitute the theme of this monograph; evaluation purpose, 
stakeholders, planning, coordination, sources, collection, analysis, 
and utilisation. In practice, most evaluation projects and reports 
focus on one or more program aspects: needs, context, goals, staff- 
ing, participation, program, materials, and outcomes. Following 
these two examples, the main section of the monograph is in eight 
parts corresponding to the guidelines, each of which contains sev- 
eral examples and brief comments that illustrate a few variations on 
the theme. Four of the guidelines pertain to leadership of an eval- 
uation effort — coordination, purpose, stakeholders, planning. 
Neither the guidelines that constitute the theme nor the aspects on 
which the variations focus are linear, but are interrelated compon- 
ents that can be addressed in any sequence. 

The first example is based on a rationale for evaluation of adult 
basic education, which can be readily adapted for use with various 
types of educational programs for adults (Grotelueschcn, Gooler, 
and Knox 1976). In this example, the fictional viewpoint character 
is Damon Rodrigue:. Coordination of evaluation is one of Damon’s 
responsibilities as associate director of a large urban adult basic 
education program. His basic planning strategy is to involve various 
stakeholders in multiple assessments, in an effort to increase their 
use of conclusions. 

Damon included self-assessment to enhance staff selection, devel- 
opment, and placement, which can also contribute to the validity 
of the conclusions and commitment to utilization. As with all 
assessment procedures, this had both strengths and limitations. He 
used complementary assessments such as supervisory' review, peer 
assessments, and feedback from participants. The data collection 
piocess was aided by explicit assessment criteria and a comparative 
database from similar programs. Because teamwork was very' im- 
portant, he conducted comprehensive performance reviews of 
various staff roles in recognition of multiple influences on staff 
performance. 



Theme and Variations 



His assessment of teacher performance and development included 
various types and methods ol data collection. The purposes for 
teacher assessment were to improve instruction, reward excellent 
performance, modify' responsibilities, and promote staff develop- 
ment. He tried to separate evaluation for accountability and for 
improvement, and he recognized various aspects of performance to 
he assessed and ways of doing so. He reviewed several types of items 
for self-assessment forms and for guiding observations of interviews 
with instructors. 

One was a general form for an external evaluator to record ratings 
of the extent of supporting evidence along with comments on com- 
mitment' to program goals, sensitivity to ethnicity, dedication, per- 
sonality’ tiaits, understanding of participant background, subject 
matter competence, flexibility, and teamwork. Findings were used 
to assess program goals and teacher needs. Also, teachers might 
informally ask participants questions regarding their satisfaction 
with the instruction, such as teaching methods and materials that 
help most, those that are least helpful, and suggestions for better 
types of help. 

Another method of evaluation was a self-assessment form with 
ratings on satisfaction with the following: uses a variety of methods, 
is prepared and organized, is enthusiastic, encourages active partici- 
pation, takes teaching seriously, provides thorough coverage of 
content, strives for mastery learning, and optimizes learning pace. 

The resulting evaluation of staff development assessed their edu- 
cational needs, views of staff development plans, satisfaction with 
results and benefits, and resource support for staff development. He 
asked staff members to indicate their general role but otherwise 
respond anonymously to open-ended questions about aspects of 
their responsibilities on which they would like staff development 
help, topics of interest, preferred timing, and anticipated benefits. 
He and other staff development planners used a listing of responses 
grouped by role, to appreciate the variety of needs expressed and to 
decide on more structured needs assessments and likely types of 
inservice activities. A more structured method was a listing of po- 
tential staff development topics, which staff members could rate 
regarding their relative interest. Topics included understanding 
participants, selecting and using materials, evaluating participant 
progress, working with volunteers, improving interpersonal rela- 
tions, and varying teaching styles. 
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A follow-up opinionnaire on staff expectations enabled them to 
indicate their objectives for participating in a staff development 
activity, aspects of the activity that contributed to and detracted 
from their achievement of their objectives, the extent to which 
their expectations changed as a result of the activity, and the 
extent to which their objectives were achieved. 

A follow-up assessment of outcomes after staff members had a 
chance to use what they learned included both a rating of extent of 
impact and explanatory comments regarding the following aspects: 
enhanced knowledge, insights on how to improve practice, greater 
appreciation of an aspect of teaching, actual use of new ideas, in- 
creased networking with other staff, and situational influences that 
helped or hindered application. 

Data collection procedures related to staffing using multiple assess- 
ments typically reflect the relative emphasis on performance review 
or staff development. Goals of teacher assessment can include help- 
ing improve instruction, rewarding excellent performance, modi- 
fying responsibilities, and promoting staff development. Various 
items could be used on self-assessment forms and for guiding obser- 
vations or interviews with instructors. 

Once preliminary staff development goals and activities were draft- 
ed, staff members rated each goal regarding the importance of the 
goal to them, the desirability of the proposed activity to achieve the 
goals, their current proficiency related to the goal, and the likely 
benefits from achieving the goal. 

Another approach was assessment of teacher satisfaction with the 
stafr . J evelopment procedures they experienced in such areas as 
usefulness of materials, quality of instruction, appropriateness of 
methods, responsiveness of activities to staff concerns, thorough- 
ness of coverage, pacing of instruction, and sequencing of activities. 

Analysis of data regarding staff assessment entailed reviewing the 
generalizations that emerged from each of the foregoing multiple 
assessments, to identify the main conclusions that were common to 
many of the separate assessments and warranted attention. The 
details from the separate assessments were then used to formulate a 
plan of action. 

The second introductory example that illustrates additional con- 
cepts and procedures related to the basic theme ot program eval- 
uation is based on an evaluation report by Kiernan and Brown 
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(1992). The fictional viewpoint character is Marion Ackerman, 
who worked for a Cooperative Extension health education program 
for working women. When planning this program review, Marion 
recognized that the breadth of such program reviews (such as self- 
studies followed by site visits) makes the evaluation design and 
process an especially important influence on utilization. Illustrative 
influences on use include attention to program issues, clear evalua- 
tion purpose, and involvement of stakeholders. Also related to her 
coordination responsibility for a major program review was pro- 
vision of resources and commitment to implementation of 
recommendations. 

Marion’s approach illustrates a combination of formative and sum- 
mative evaluation to ensure that program evaluation would be 
beneficial. The purpose of the formative evaluation was to increase 
the capacity of the extension education program to benefit partici- 
pants, to enable staff to evaluate impact, and to identify' barriers* 
The program consisted of four home study lessons provided at work 
and child care sites on osteoporosis risk reduction for working 
women. Formative evaluation findings were used to modify' sites, 
marketing, program methods, and evaluation instruments. The 
staff concluded that formative evaluation was crucial for program 
implementation. The summative evaluation’s purpose was to in- 
crease accountability 1 . Following are the evaluation objectives and 
the variety of evaluation designs and data collection methods that 
were used: 

• To decide on sites for reaching working women, ages 21-34, 
who were raising children (comparison group design and data 
collection methods, including demographic information from 
enrollment forms, and feedback from participants and onsite 
program managers). 

• To assess the appeal of written materials and of two methods of 
learning, one that involved just reading the materials and one 
that included a brief motivational meeting (comparison group 
design regarding two methods, using response sheets for each 
lesson for participants to record their reactions, four focus 
groups — each with a sample of participants following the series 
of lessons, and a meeting with the group of program managers 
who presented the lessons). 

• To assess program impact on participants (participants com- 
pleted a pro- and posttest to assess knowledge, attitudes, and 
behavior changes). 
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Theme and Variations 

• To refine evaluation procedures (analyzed demographic vari- 
ables, completion rates, errors in pre- and posttest scores, and 
comments from focus groups.) 

These two introductory' examples were more comprehensive eval- 
uations than practitioners usually conduct. This was because re- 
ports of more modest evaluations are seldom published, and be- 
cause these two comprehensive evaluations illustrate all of the 
following eight guidelines. These guidelines each reflect useful 
concepts and procedures for planning and conducting evaluations; 
they constitute interrelated components and are not proposed as a 
linear process of steps to be followed. After each guideline there are 
comments regarding concepts and procedures Damon and Marion 
used in the two case examples. 

1. Coordination. The extent and type of coordination that is desirable vanes with the size 
of the effort, the number of people involved, the program aspect to be conducted, and the 
extent of teamwork required, but for large projects it is desirable to designate a 
coordinator. 

a. Damon coordinated the adult basic education evaluation with the help of an external 
evaluator. Because of the staffing focus, coordination emphasized teamwork. 

b. Marion coordinated a comprehensive review of an Extension health education pro- 
gram, which included attention to resources such as facilities and financial support. 

2. Planning. Worthwhile evaluations are carefully planned, using designs, methods, and 
levels of effort that fit the evaluation purpose and resources, with guidelines that reflect 
experience, standards, publications, and consultant assistance. 

a. Damon and his evaluation planning committee arranged for performance review of 
people in various staff roles and prepared assessment criteria for use when collecting 
and analyzing data. 

b. Marion arranged for cooperation from participants and instructors and for focus group 
sessions with site coordinators. 

3. Purpose. Clarify the purpose of an evaluation project, to guide planning and implemen- 
tation, obtain resources and stakeholder support, and ensure that benefits from findings 
exceed time and money expense for the evaluation. 

a. When clarifying the purpose of the evaluation project, Damon distinguished between 
formative evaluation for program improvement (such as assessing instructor’s educa- 
tional needs) and summative evaluation for accountability (such as assessing goal 
achievement and program impact). His main purposes were to reward effective staff 
performance, improve instruction, and promote staff development. 

b. Marion’s intent was to use formative self-study to modify the program, with special 
attention to increased benefits related to health, appeal of instructional materials, 
identification of barriers, and assessment of impact for greater accountability'. 

Continued 
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4- Stakeholders. Early involvement of pertinent stakeholders contributes to responsiveness 
to issues they care about, useful viewpoints on program features, and their use of conclu- 
sions to implement recommendations. 

a. In addition to the participants (learners) Damon included program administrators 
and other instructors (peers) as program stakeholders involved in the evaluation. 

b. In addition to participants, Marion included program site administrators as prime 
stakeholders whose future decision making should reflect evaluation conclusions. 

5. Sources. Decisions about sources of information should reflect issues important to stake- 
holders, sources credible to them, inclusion of multiple indicators, and use of a prelimi- 
nary rev'ew to select program aspects on which to focus. 

a. Sources of evaluation information (data) that Damon used included participant feed- 
back, peer assessment, supervisory review, follow-up study, and an external evaluator 
to critique their findings. 

b. Marion used a comparative group design with different instructional materials and 
obtained feedback from participants and site coordinators. 

6. Collection. Most evaluation projects use a combination of quantitative and qualitative 
data collection procedures. 

a. Damon used instructor self-assessment forms, peer observation checklists, and inter- 
view guides. 

b. In addition to staff questionnaires, Marion used achievement tests and response 
sheets regarding participants’ reactions to materials. 

7. Analysis. In practice, analysis of evaluation data varies in formality', perspective, types of 
data, timing, and use of additional information to interpret findings. 

a. Damon’s use of multiple assessments resulted in listings of responses by roles, noting 
the variety of staff development needs that resulted, and comparisons with evaluation 
summaries from similar programs. He then had staff members review the evaluation 
findings and suggest implications for staff development. 

b. Marion’s data analysis included comparison of pre- and posttest scores, discussion of 
findings with groups of program coordinators, and meta evaluation of the evaluation 
procedures and materials that were used. 

8. Utilization. Stakeholders are encouraged to use evaluation conclusions by their involve- 
ment in the process and by the credibility of the findings, but their involvement should be 
appropriate to the nature of the evaluation project and to the backgrounds and interests 
of each category of stakeholders. 

a. Damon combined self-assessment and peer review to produce a plan of action for 
improved staff performance and development, as a strategy to encourage use of eval- 
uation conclusions. 

b. Marion combined a comparative group design to produce valid findings, with various 
types of formative evaluation to increase commitment by site coordinators and other 
stakeholders to use the evaluation conclusions. 
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la practice, few adult and continuing education program evalua- 
tion projects are as comprehensive as these two examples. The 
foregoing guidelines constitutes a basic theme for evaluation stra- 
tegies. Following are eight program aspects or decisional areas on 
which an actual evaluation project might focus. 



1. Needs. Assessing the unmet educational needs of potential participants in order to plan 
responsive programs. 

2. Context. Analysing the organizational and societal context of a program in order to take 
into account organizational priorities and resources, other providers, and pertinent 
trends. 

3. Goals. Assessing the relative desirability of alternative program goals in order to clarify 
and select those that are most important for program planning and evaluation. 

4. Staffing. Assessing expected and actual staff’ performance in order to guide staffing deci- 
sions related to individual and team performance and organizational effectiveness. 

5. Participation. Assessing initial and continued learner participation and progress as a 
developmental process that includes preparation, achievement, and benefits in order to 
guide assistance regarding assessment of prior learning style, encouragement of self- 
assessment, feedback regarding achievement, and commitment to application. 

6. Program. Comprehensive evaluation of a single course or an entire set of offerings often 
includes both self-study and external review in order to shape offerings, improve teaching, 
increase quality, and obtain accreditation. 

7. Materials. Assessment of materials effectiveness and learner satisfaction in order to se- 
lect, develop, and improve all types of educational materials. 

8. Outcomes. Conclusions from summative evaluation of outcomes, results, impact, and 
benefits that are of special interest to external stakeholders in order to undergird account- 
ability' decisions, and also important for planning and improvement decisions by internal 
stakeholders. 





Application of 
Guidelines 

This chapter reviews literature on each of the guidelines that con- 
stitute the basic evaluation theme of this monograph. Each of the 
eight parts of this chapter focuses on a guideline by presenting sev- 
eral examples that illustrate variations on the basic evaluation 
theme. (For instance, the part on evaluation purpose begins with 
examples that focus on staffing and goals, whereas the part on 
planning begins with examples that focus on participation and 
outcomes.) Each part concludes with a review of writings and 
concepts that address additional variations such as needs assess- 
ment and contextual analysis. 



Purpose — 

Clarifying the purpose of any evaluation activity helps ensure that 
the evaluation will be cost beneficial. Efficiency and utilisation can 
be increased by focusing on issues important to the relevant stake- 
holders. An evaluation activity can be initiated by people associ- 
ated with the provider agency, or it can be in response to an ex- 
ternal influence. Also, evaluation conclusions from other program 
aspects can contribute to planning, conducting, and interpreting an 
evaluation (Steele 19S9). 

Clarification of the purpose of a statewide evaluation of workplace 
education programs was enhanced by 1-day workshops on the topic 
attended by program administrators (Jurmo 1993). The initial 
workshops in various parts of the state, and a follow-up workshop, 
helped the participating administrators work in teams to develop 
evaluation guidelines for subsequent use when other people were 
making decisions about funding their proposals. The specific pur- 
poses were to — 



• Enable participants to develop evaluation teams in their respec- 
tive programs 

• Enable participants and their teams to collect evaluation infor- 
mation for use in preparation nr action plans tor program 
improvement 

• Enable participants and their team'' to develop effective pro- 
posals for external funding 
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• Develop a corps of practitioners experienced in effective eval- 
uation strategies and methods 

These workshops and related activities resulted in capacity building 
for practitioners regarding evaluation generally, but focused on the 
guideline regarding clarification of evaluation purposes and the 
staffing aspect. 

A second example ot an evaluation approach that emphasizes clari- 
fication of the purpose of the evaluation occurred in a Cooperative 
Extension program. Bennett and Rockwell (1995) emphasize that 
societal conditions can serve as a major basis for assessment goals. 
They present an integrated approach to planning and evaluation 
titled Targeting Outcomes of Programs in which societal conditions 
are a focus for assessment related to both program development 
and impact evaluation. Assessment of goals considers the priority of 
societal needs and opportunities regarding importance and the like- 
lihood that specific conditions will be improved through extension 
programs. By envisioning desired conditions and identifying dis- 
crepancies with current conditions, specific practices can some- 
times be identified that would help achieve such desired condi- 
tions. Related practice adoption is among the highest aims of ex- 
tension programs. Assessment questions related to goals include — 

• What present conditions will the program help improve? 

• What describes the situation once the condition has been 
improved? 

• How will you know it the condition has been improved? 

• How will your program address public interests but also con- 
sider the private needs ot individuals? 

Responses to these questions could be used to decide on the rela- 
tive importance of various societal conditions that an educational 
program for adults might address through both program planning 
and evaluation impact. Clarification of the evaluation purpose is 
also an important guideline for assessment of other program as- 
pects. Following are some brief examples related to needs assess- 
ment, contextual analysis, and program review with references' to 
sources of additional details. 

Because the concept ot needs assessment is often ambiguous, atten- 
tion to the purpose ot this type ot evaluation is especially helpful. 
The recent book bv Queenev (1995) provides a useful overview 
and rationale for the range ot needs assessment procedures, such a> 
focus groups, kev informants, supervisor evaluations, nominal 



14 



1J 



Application of Guidelines 



group, surveys, performance assessments, simulations, and assess- 
ment centers. Clarification of the evaluation (needs assessment) 
purpose can guide selection of procedures that can best serve the 
purpose. 

Contextual analysis is another aspect of evaluation for which clari- 
fication of the evaluation purpose is especially valuable. A form of 
contextual analysis can be used in a workplace setting to decide 
whether a human resource development program is warranted. For 
example, the following evaluation procedures might be used for 
contextual analysis: 

* Action research studies combine inquiry with action to solve a 
problem in ways that have direct application in the work set- 
ting. For example, such a project could pertain to collaboration 
and ways to assess its effectiveness. (See Quigley and Kuhne 
1997 for rationale on action research.) 

* Case studies typically use various data collection methods to 
assemble both qualitative and quantitative data to understand 
specific groups or events that are complex and highly influ- 
enced by the context. Benchmarking the practice of similar 
providers for comparative analysis related to an accreditation 
self-study is an example. 

* Expert reviews occur when the opinions of people with excep- 
tional expertise are obtained through correspondence, ques- 
tionnaires, personal or telephone interviews, panel discussion, 
or Delphi Technique (successive rounds of refinement of pri- 
orities or a rationale by an expert panel). Expert review is espe- 
cially appropriate for identifying emerging trends, such as tech- 
nological developments or public policies that have direct im- 
plications for responsive educational programs. 

* Literature reviews can be efficient procedures for identifying 
key traits and characteristics related to work performance. 
Database searches can identify relevant reports (Brinkerhoff 
1987). 

The breadth of program reviews makes clarification of the evalua- 
tion purpose an essential guideline. The educational program (con- 
tent and methods) is a usual focus of program evaluation and a 
major influence on participant achievement. The program to be 
assessed can be a single course or workshop* or the provider agen- 
cy’s entire range of offerings for a Year. The evaluation conclusions 
can be used to improve the fit among program characteristics such 
as goals, content, learners, instructor, methods, and resources 
(Caffarella 1994; Deshler 1984; Rothwell and Cookson 1997; 
Worthen, Sanders, and Fitzpatrick 1997). Among the reasons tor 
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program review are shaping offerings, accreditation, increasing 
quality, feedback to participants, improving teaching, and embrac- 
ing reflective thinking. 

Assessment can help clarify expectations, encourage autonomy, 
deepen understanding, diagnos* misunderstandings, and judge 
achievement. For instance, an anatomy instructor used evaluation 
procedures that encourage participants to demonstrate their under- 
standing and discourage superficial approaches (Ramsden 1992). 

He did not use multiple choice questions that might reinforce 
remembering unconnected facts. Instead he used brief essay ques- 
tions and oral dialogues to encourage learners to explain important 
principles, subsuming descriptive information. In this way, the 
teaching function of assessment took first priority. 

In another instance, an interior design instructor used personal 
consultations and other forms of assessment to help participants 
identify important concepts and do their best, by understanding the 
problems they were experiencing and to suggest alternative direc- 
tions. Assessment procedures included asking participants to iden- 
tify information necessary to translate design into reality, establish 
an accessible format, evaluate the information, and present a plan 
concisely as a coherent argument (ibid.). 

A number of additional publications are valuable for evaluation of 
other program aspects. Overviews of needs assessment are provided 
by Pennington (1980) and by Johnson, Meiller, Miller, and Sum- 
mers (1987). Cassara (1990) and Ross-Gordon, Martin, and 
Briscoe (1990) focus on minorities. Examples of contrasting needs 
assessment purposes and approaches include Naishitt and Abur- 
dene (1990), Pigg, Carrier, and McDonald (1995), and Sisscl 
(1996). Additional publications that illustrate evaluation purposes 
related to workplace education include Bernthal (1995), Grubb 
(1995), Harri-Augstein and Webb (1995), and Mikulecky and 
Lloyd (1995a). Examples regarding other types of programs include 
Kerka (1995) on correctional education and Young 1995) on lib- 
eral education. 
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Summary' 

Clarification of the purpose of an evaluation is worthwhile. Such clarification may be 
straightforward when instructors are assessing aspects of their individual courses, but the 
purpose may be unclear when an entire set of program offerings arc evaluated or when an 
evaluation report is required as a condition of external funding. Involving stakeholders in 
planning and conducting an evaluation can focus it on expectations and benefits that they 
value, respond to both educational needs and social conditions, and contribute to institution 
building and use of evaluation conclusions. A focused evaluation purpose can ensure that the 
benefits of the evaluation warrant the costs, and it can guide the selection of evaluation 
procedures that fit the evaluation purpose and program characteristics. 

Stakeholders 

A crucial consideration in effective program evaluation is the in- 
volvement of stakeholders. Stakeholders include participants, 
instructors, coordinators, policy makers, accreditors, and funders. 

Such involvement can occur for prospective, current, and former 
participants. Early involvement by stakeholders in planning an 
evaluation contributes to decisions on the extent of effort and 
stakeholder involvement, the responsiveness of the evaluation to 
issues important to stakeholders, useful viewpoints on program 
features, and use of the conclusions to implement recommenda- 
tions. Leadership on behalf of evaluation should give equal atten- 
tion to technical procedures related to data collection and analysis 
and to interpersonal relations to obtain and maintain cooperation 
(Patton 1986). 

Because multiple external stakeholders are typically related to con- 
textual analysis, their involvement in such an evaluation is espe- 
cially important (Knox and Associates 1980). For example, con- 
textual analysis was a major part of a comprehensive analysis of a 
pilot community arts project. In 1986, a visual artist served for 4 
weeks as a community arts developer in southeastern Colorado, 
funded by the state council on arts and humanities. Resources for 
the evaluation, conducted by Jones (1988), included staff support, 
a minimal budget, and some in-kind contributions. 

The program goals were that the artist and community' should he 
mutually challenged and enriched, that the artist serve as a cata- 
lyst, and that the arts contribute to community' development. The 
evaluators looked for possible change among multiple social systems 
including the artist, arts council, local artists, and the community. 



O ) 



17 





Application of Guidelines 



The evaluation included triangulation using six approaches; inter- 
views with key participants, surveys of representative households 
before and after the residency, informal interviews on the street, 
direct observations of arts activities, a short questionnaire left in 
local businesses, and monitoring of the local newspaper. The eval- 
uators concluded that all systems were affected. The evaluation 
report indicated that a better understanding of a community devel- 
opment project can be gained from a modest investment in evalua- 
tion (ibid.). Findings from evaluation of such a pilot project can be 
very useful for addressing likely situational and environmental in- 
fluences in subsequent program offerings. 

Stakeholder involvement is especially important for program re- 
views. Most evaluations of entire educational programs include 
various components and stakeholders. Because of the resulting 
complexity, interpersonal relations to win and maintain coopera- 
tion from multiple stakeholders are a major influence on the suc- 
cess of the evaluation (Patton 1986). Sometimes a program review 
includes a self-study by internal stakeholders and a site visit on 
behalf of external stakeholders. Quality assessment typically in- 
cludes various stakeholders. For example, the Minnesota Extension 
Service decided to address quality improvement by use of a ques- 
tionnaire to assess quality by comparing expectations of partici- 
pants and staff with perceptions of performance (Chen, Krueger, 
and Leske 1993). The evaluation focused on two gaps: between 
participants’ expectations and staff perceptions of participants’ 
expectations and between participants’ expectations and their per- 
ceptions of received service. The evaluation procedures included 
sampling, instrumentation, administration, and analysis. Question- 
naires were sent to Extension staff and to participants on county 
extension mailing lists. 

Return rates were as follows: 92 percent of 261 county educators, 
85 percent of 19 administrators, 86 percent of university depart- 
ment heads, and 66 percent of 500 participants. The modified 
instrument was well received in instrument development and field 
testing, indicating acceptable face validity. Although there were 
statistically significant discrepancies between staff perception of 
participant expectations of service qualiry and participants' actual 
expectations of service, the differences were small and not of prac- 
tical significance. The participants’ expectations of excellent ser- 
vice were higher than their perceptions of received service for al- 
most all questions, but the differences for only three items were of 
practical significance (responding quickly to community crisis, pro- 
viding current and accurate research based information, and in- 
forming participants about service). 
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Involvement of stakeholders is also an important guideline for 
assessment of other program aspects. Following are some brief 
examples related to goals, participation, and outcomes, with refer- 
ences to sources of additional details. 

When assessing program goals, stakeholders can assist in recogniz- 
ing the influence of various stakeholders on program priorities, 
considering the range of stakeholders who contribute to goal set- 
ting, and implementation of recommendations (Patton 1986). For 
example, an affinity process can be used to seek agreement on 
goals. When stakeholders have diverse ideas about program' goals, 
an affinity diagram process can be used to identify major themes 
that connect related ideas, with a minimum of discussion. The 
basic process is as follows: assemble appropriate stakeholders; select 
an important issue for resolution; state the issue generally to en- 
courage ideas about future goals; and brainstorm many ideas on 
Post-it^ slips until there are no additional suggestions. Place Post- 
it lv slips for each suggestion on several large sheets of paper. Mem- 
bers can help group similar Post-its s . Continue until group mem- 
bers are satisfied with the groupings. Finally, write a concise phrase 
that summarizes the meaning of each set of Post-its which is suf- 
ficiently clear that people who did not participate will understand 
it, and place the phrase at the head of the set of Post-its K . 

Another example of involvement of stakeholders is action research 
in practice settings, which can contribute to participation assess- 
ment. Action research uses evaluation as an ongoing part of a cycle 
of problem posing and problem solving to achieve program im- 
provement (Quigley and Kuhne 1997). Practitioners both guide 
the action research and use the conclusions. Developing an action 
research project can be conceptualized in three phases: planning, 
action, and reflection (each of which includes evaluation). The 
planning phase includes the first three steps. Step one is under- 
standing the problem (state briefly why it exists, prior students, 
worth working on, evident causes, initial ideas). Step two is de- 
fining the project (proposed project, when to begin, managing 
process and results, needed materials, needed approvals, gaining 
cooperation, who will help assess). Step three is selecting the mea- 
sures (specify baseline and criteria for success, timeline for evalua- 
tion, methods for data collection, potential deterrents). The action 
phase (step four) is the implementation of an action and observa- 
tion of the results (following plan, data collection that closely 
monitors progress, continued cooperation, data summary) - The 
reflection phase includes steps five and six. Step five is evaluating 
the results (conclusions about the problem and intervention, 
relation to criteria for success, opinions about the project). Step six 
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is reflecting on the project (replication, a second or third evcle of 
the project). The action research approach illustrates the manv 
connections between evaluation and participation in staff develop- 
ment and program improvement, which extends from initial en- 
gagement through learning activities to application. 

A third example of involving stakeholders pertains to assessment of 
outcomes. Such impact evaluations sometimes pertain only to im- 
proved performance by individual participants as they apply what 
they learn. In such instances they can icport on the benefits rhev 
receive. However, some impact evaluations also include assessment 
of benefits reported by other people associated with participants in 
group, organizational, or community settings. Their early and ap- 
propriate involvement in outcomes evaluation can both strengthen 
the assessment and enhance their use of conclusions. Assessing 
benefits to such stakeholders can be challenging, hut can become 
more feasible by attention to comparison group, elapsed time, pre- 
post measures, time series, costs and benefits, and use of evidence. 
Sometimes, the evaluation goal allows a simple procedure, such as 
sending a questionnaire regarding general satisfaction with program 
elements to participants, instructors, coordinators, work supervis- 
ors, and community organization leaders (Grotclueschen, Cooler, 
and Knox 1976). 

Additional examples can illuminate ways to involve stakeholders 
when the focus is on other program aspects: Rothwell and Cookson 
(1997) on self-assessment by instructors, Brinkcrhoff ( 1987 ) on 
assessment of participants and people in related roles, Braskamp 
and Ory (1994) on staffing assessment, Caffarella (1994, pp. 1 28- 
132) on a participant questionnaire as part of a program review, 
and Knox (1979b) on assessment of impact. Additional examples 
of evaluation that is responsive to stakeholders include St. Pierre er 
al. (1995) on family literacy, Dodrill ( 1995 ) on teleconferencing, 
and Wagner ( 1995 ) on distance education. 



Summary 

Early involvement of multiple stakeholders in evaluation can contribute to decisions about 
the extent of their involvement, responsiveness to their viewpoints, and wavs to encourage 
their use of conclusions. More comprehensive evaluations entail multiple social systems and 
both internal (staff) and external (funders) stakeholders. Evaluation leaders should devote 
equivalent attention to t .chnical procedures and achieving cooperation. Stakeholders may be 
interested in benefits at individual, group, organizational or community levels. Action 
research constitutes a major evaluation strategy. 
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Plannings 

Tv Really, worthwhile evaluations are carefully planned, using 
designs, methods, and levels of effort that fit the evaluation purpose 
and resources. Guidelines for planning a specific evaluation can 
come from various sources including experience, standards, publi- 
cations, and consultant assistance (Patton 1986; Sanders 1994; 
Worthen and Sanders 1987; Worthen, Sanders, and Fitzpatrick 
1997). 

Actual planning procedures reflect the contingencies of the specific 
situation such as program history, quality, stakeholder support, 
available resources for evaluation, evaluation expertise, and con- 
cern about program improvement. The following example illu- 
strates planning of an evaluation project in order to obtain findings 
to enhance learner enrollment, persistence, reflection, achieve- 
ment, and application. 

Two consecutive demonstration projects on education and family 
literacy instruction were evaluated regarding process and impact, 
with findings relevant to encouraging participation (Garcia, 

Hasson, and Le Blanc 1997; Garcia, Hasson, and Younkin 1992). 
The objectives were to improve the English language proficiency of 
limited English proficient Hispanic parents and their elementary 
students, increase their adjustment to a new society, and enable 
parents to become more active in their children’s education. The 
60-7C hour program included instruction related to literacy, life 
skills, parenting, school involvement, and communication. 

In the 1989-92 project, evaluation included review of program and 
ongoing evaluation records, interviews with project staff, a demo- 
graphic questionnaire, pre/posttests, and parent surveys. Specific 
instruments included life skills achievement, literacy placement, 
civics inventory, parent understanding of schools, and parent sur- 
vey of parent involvement with their child’s educational progress. 
Evaluation questions pertained to program process and to signifi- 
cant increases in literacy, parenting, civic involvement, and parent 
involvement. 

The 1992-96 project was similar and built on the previous one. 
Placement and assessment instruments were developed and field 
tested, including criterion-referenced testing and portfolio assess- 
ment. The evaluation report described the participants’ back- 
grounds, and reported significant improvements on all indicators of 
parent involvement and language development. Project materials 




21 



Application of Guidelines 



and systematic planning were made available for use elsewhere. 

The evaluation part of the project guided program decisions, in- 
creased program responsiveness, and contributed to the applica- 
biliry of the program and materials elsewhere. 

Systematic planning is especially important tor major evaluation 
studies of outcomes. Such impact assessments can be for purposes 
of accountability, improvement, or planning and can focus on any 
combination of participants, instructors, topics, or contents. An 
important early planning decision is the general evaluation design 
or approach to use. Alternative approaches include nonexpert 
mental (self-study, external review), quasi-experimental (time 
series, comparison group), and experimental (random assignment 
of participants to treatments). Possible designs include objectives- 
based, case study, systems analysis, expert review, follow-up, cost 
benefit, and analysis of practice adoption (familiar in Cooperative 
Extension programs). Planning for a specific outcome evaluation 
design tends to reflect attention to benefits to individuals, groups, 
organizations, or communities. 

An impact assessment of Cooperative Extension programs illu- 
strates an approach to planning an outcomes assessment (Forest 
and Marshall 1978). A representative Wisconsin county was se- 
lected for an impact study, in which local efforts were supplement- 
ed by state staff and contracted telephone interviewing. Local 
stakeholders were involved to increase understanding, cooperation, 
and use of conclusions. The impact assessment covered all aspects 
of Cooperative Extension programs in the county. Many additional 
sources of data were used: reports, documents, and records cover- 
ing several years. Several hundred leaders and 1,000 residents were 
interviewed, with the conviction that perceived value based on 
multiple contacts is understandable and relevant to Extension deci- 
sions, and phone interviews are much less expensive than observa- 
tion and personal interviews. Interviews covered levels 3-7 of the 
Bennett and Rockwell (1995) hierarchy: participation, reactions, 
achievement, practice, and end results. 

Improvements to which extension was reported to have contri- 
buted pertained to economics, government, health, abilities, envir- 
onment, and educational opportunities. Before reporting on find- 
ings, groups of stakeholders were asked to establish standards that 
should be attained by extension programs. This increased their 
interest in interpretation of findings and allowed comparison of 
actual impact with desired impact. Extension staff and leaders in 
Wisconsin and other states were asked no evaluate the evaluation 
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and indicate how they used the reports. All of these features re- 
flected the initial plans (Forest 1979; Forest and Marshall 1978). 
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Planning considerations vary with other program aspects. Following 
are some examples related to needs assessment, program review, 
and outcomes. Although much needs assessment is informal, it 
tends to be more useful when the process, evidence, and conclu- 
sions are made more explicit and when several types of evidence 
are included to provide some cross-validation. Planning can be 
strengthened by considering various needs assessment procedures 
and selecting those best suited to the situation (Queeney 1995). 
More formal needs assessments are preferable when there are un- 
familiar participants and objectives. Decisions about the extent of a 
needs assessment effort reflect available resources and preferred 
procedures. 

Some needs assessment plans can be quite modest. For example, 
one of the most valuable types of information for an instructor is 
about the learners’ current proficiency regarding the content. Brief 
diagnostic inventories such as multiple choice questions and open- 
ended questions can be completed at the beginning of a unit. It 
previews for participants the main concepts to be learned and for 
the instructor it helps specify the range of participant backgrounds, 
the most helpful starting points, and a baseline for assessing sub- 
sequent achievement. Their responses can be grouped by level of 
current knowledge. A summary of responses can be shared with 
participants and they can meet in small representative groups to 
agree cooperatively on correct answers (Angelo and Cross 1993). 

Another method is assessment of unclear concepts, as an educa- 
tional program progresses, in order to discover efficiently the deas 
that participants found unclear, confusing, or hard to understand 
so that the instructor can address these emerging needs in the next 
program segment. Following a reading, discussion, presentation, or 
case analysis, participants are asked to write in a few words or 
phrases the main points or ideas that they found most “muddy” or 
unclear. Tins technique is especially useful after presenting large 
amounts of new information. The instructor can read the written 
comments and sort them in piles of related comments. This sum- 
mary can help the instructor devote some time to clarifying the 
most confusing concepts, in the next program segment (ibid.). 

Planning for a program review is usually tor the purpose of improv- 
ing the fit among program characteristics such as goals, scope, 
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content, learners, instructors, methods, and resources. A prelimi- 
nary review of such characteristics can guide specification of the 
issues on which to focus an evaluation project. 

An example of planning a program assessment occurred in a Suiv 
belt state’s Cooperative Extension Service in which turfgrass for 
golf courses and residents was a major economic consideration 
(Wright 1984). A county extension adviser with this specialty 
developed a monitoring system that served several purposes. One 
was to obtain information from the professional turf managers and 
homeowners who participated in the turfgrass education program 
regarding the extent to which they were following recommenda- 
tions. Information was available from copies of soil test reports that 
were sent routinely. Additional sources of monitoring information 
came from phone inquiries to Extension regarding pertinent prob- 
lems and phone interviews by the adviser and members of a turf 
grass committee, with a sample of people who had their soil tested 
recently. Most of the monitoring information was from outside 
Extension, and included program impact and environmental influ- 
ences. Considerations related to use of monitoring systems include 
adaptability, burden, and resources. The utility of monitoring is 
related to routine data collection and use of conclusions. The pro- 
cess can be of special benefit to practitioners who learn about pro- 
gram functioning. 

In contrast to more major impact studies with extensive resources, 
sometimes an instructor wants to evaluate the outcomes of a course 
or workshop in order to both assess impact and guide improve- 
ments. For instance, evaluation of the results and benefits of the 
program and its components might include the following planning 
procedures: 

• Review expected outcomes and satisfaction with methods, to 
clarify the rationale for both the evaluation and use of 
conclusions. 

• Locate reports on pertinent studies with similar outcomes and 
methods, to note findings and evaluation procedures. 

• Review the resulting overview, and it it is sufficient for decision 
making use it, but if not, proceed with the evaluation. 

• Decide on summative evaluation goals, indicators of impact, 
and preliminary rationale for relations between methods and 
outcomes. 

• Characterize alternative methods to be studied. 

• Describe content, learners, and context related to methods. 

• Design a basic and efficient impact study likely to answer the 
main questions about outcomes. 
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• Use the preliminary review of past experience and literature, 
and a pilot study, to conduct the outcomes assessment, and 
then use findings to improve instruction with formative eval- 
nation to guide ongoing improvement. 

• Collect data, using existing instruments where satisfactory and 
perhaps develop valid instruments and procedures that do not 
disrupt the program. 

• Analyze data, by comparing method effectiveness in relation to 
indices of impact. 

• Draw implications for improvement. 

An even simpler assessment of impact could he related to partici- 
pant use of a work-related learning plan to guide individualization. 
Participants might help plan a results evaluation related to some 
shared goals, which might include a set of rating scales on extent of 
implementation as viewed by participants, peers, and supervisors. 
Participants who use these rating scales in parallel evaluations can 
compare their findings and then discuss variation in program suc- 
cess and related influences. 

A follow-up study can be combined with participants’ early written 
projection of their expected outcomes, related to implementation. 
After the program and sufficient elapsed time, a copy of their ex- 
pected use of what they learn could be sent to participants along 
with a reply form on positive and negative results of their imple- 
mentation efforts. Their willingness to respond might be enhanced 
by the prospect of receiving a summary of implementation experi- 
ences of all those who respond (Knox 1986). 

Planning evaluations related to other program aspects reflects vari- 
ations related to such focus. For example, contextual analysis can 
be very broad to include attention to societal influences and other 
providers, so standards and examples of past contextual analysis 
can be especially helpful (Forest and Marshall 1978; Joint Com- 
mittee on Standards for Educational Evaluation 1994). Careful 
planning is essential for successful use of an assessment center for 
evaluation of participation (Queeney 1995). A rationale should 
guide evaluation and improvement of educational materials (Flagg 
1990). Askov, Van Horn, and Carman (1997) recommend using 
guidelines, involving participants, and using appropriate procedures 
for adult basic education participant assessment. Follow-up audits 
can be used to assess the impact of enterprise skill training 
(Brinkerhoff 1987). 

Examples that illustrate various forms of evaluation planning 
include the following: “HRD Evaluation Issues” (199 6),J aim son 
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(1997), Mikulecky and Lloyd (1995b), and Shapiro ( L995) on 
workplace education; Bruder (1997) on medical education; Cele* 
brating Excellence (1995) on alternative degree programs; Russon 
ct al. (1995) on Extension programs, and Ziegler and Sussman 
(1996) on literacy. 



Summary 

The foregoing examples illustrate planning for evaluation projects that vary greatly in purpose 
and scale. Shared features of effective planning include — 

• Matching project scale to purpose and resources 

• Building on related reports and expertise 

• Reflecting local contingencies in evaluation plans 

• Involving stakeholders in evaluation planning and interpretation to encourage use of 
conclusions 

• Using sound evaluation design and procedures to produce valid findings 

• Building on previous evaluations of the same program 



CoordinatioiL 



Coordination of an evaluation project varies with the size of the 
effort, the number of people involved, and program aspects to he 
evaluated. However, there are some basic ideas about managing an 
assessment that apply more generally. Even modest evaluations 
benefit from teamwork, but for larger and more complex assess- 
ment projects, it is desirable to designate someone to coordinate 
the effort. Following are some suggested guidelines for coordination 
that results in sound planning and implementation (Grotelueschen 
1980): 

• Seek agreement on the main purpose and focus of the 
evaluation. 

• Keep the evaluation project as small and manageable as possi- 
ble, consistent with the purpose. 

• Recognize the importance ot both valid information and sound 
interpretations. 

• Share examples to illustrate concepts and procedures. 

• Involve and obtain cooperation from stakeholders by being 
responsive to issues important to them. 

• Identify and obtain sufficient expertise and time devoted to 
evaluation. 

• Obtain necessary' resources. 
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• Encourage use of conclusions and recommendations, by atten- 
tion to benefits, facilitators, and barriers. 

• Recognize and address situational influences. 

• Use existing information when feasible. 

• Minimize disruption to the program. 

• Use prior agreements and negotiation to minimize conflict. 

The following two examples illustrate coordination concepts and 
procedures for evaluation projects focused on needs assessment and 
on outcomes. Many types of educational needs assessment can be 
conducted by an instructor or coordinator with little additional 
assistance. Familiarity with evaluation standards (utility, feasibility, 
propriety, accuracy) can enhance small-scale needs assessments. A 
coordinator can also encourage continuous assessment to monitor 
responsiveness to learner needs. 

For larger needs assessments for which coordination is important, a 
collaborative approach can be beneficial. Such a collaborative ap- 
proach is illustrated by the following plan for conducting a needs 
assessment (Green and Associates 1984). The chair of a health 
occupations association committee on continuing education initi- 
ated a joint effort with the director of a university center engaged 
in continuing education research and outreach. The association 
had concluded that conducting a needs assessment was of high 
priority for planning continuing professional education activities for 
members of the association and of related health care occupations. 
There was an early commitment to collect data from various 
sources in order to produce findings relevant to the objectives and 
activities of educational activities by various continuing education 
providers that entailed various ways of learning. The joint coordi- 
nation by an association committee chair and a university center 
director contributed to a needs assessment useful to both asso- 
ciation members and practitioners in related health occupations, 
any of whom might be served by various continuing education 
providers. 

The needs assessment plan that they prepared specified the contri- 
butions they and other people would make to conducting the needs 
assessment. Included were the intended recipients of the needs 
assessment report (members ot the health occupations, association 
leaders, educators, and potential providers of continuing educa- 
tion) and the types of decisions they might make (priorities regard- 
ing topics, learning methods, special emphases, and attention to 
both acquisition and application of enhanced proficiencies). The 
plan also included data collection by mailed questionnaire to mem- 
bers of the association and related occupations, regarding their 
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characteristics, past educational participation, interest in topics and 
arrangements, and perceived facilitators and barriers to participa- 
tion. In addition to interviews with members and others, and data 
analysis procedures, the plan included specifying the recipients of 
the report, encouraging completion of the questionnaire, and 
encouraging dissemination and use of finding to improve continu- 
ing education opportunities. Coordination of such an evaluation 
project is important at planning, implementation, and utilization 
stages. 



Coordination of an impact study is typically complex, as reflected 
in the following example related to continuing medical education. 
The level of educational effort that it takes to produce a desired 
impact is one conclusion that can result from evaluation of out- 
comes. Coordination of this project contributed to convincing 
conclusions. A 4-year study of educational interventions with 
1,000 primary' care physicians at multiple sites in Michigan ex- 
plored the impact of voluntary action-oriented continuing medical 
education on improvement of ambulatory' medical care delivery 
(Knox 1979b; Payne ct al. 1978). The essence of the educational 
intervention was that physicians must perceive recommended 
changes as useful, there must be sufficient encouragement of 
changes to sustain initial commitment to change, and there must 
be sufficient time for the changes to he made and stabilized. 

The first level of intervention was minimal, almost a control group, 
which consisted of dissemination of information about effective 
performance to the physicians. The second level consisted of sev- 
eral dav-and-a-half workshops that included discussion of problem- 
solving procedures and implications for planned change. The third 
level added consultations with hospital administrators to help plan 
ways to increase staff involvement in making clinical changes and 
management of ambulatory services. The first level (reporting) 
intervention produced minimal change in performance. The sec- 
ond level (seminar) intervention produced significant improve- 
ments. The addition of a third-level intervention almost doubled 
the improvement. The coordination of such field trials typicallv 
entails preparation of a detailed plan or proposal, acquisition of 
external funding, arrangements tor learner participation in the 
alternative educational activities, collection and analysis of data 
regarding educational process and outcomes, and reporting. 

Usually, support for such an evaluation is an adjunct to a larger 
demonstration project. 

Variations on the coordination theme are reflected in the following 
examples related to contextual analysis, goals, and program review. 
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Coordination of contextual analysis (usually by a program adminis- 
trator) can be enhanced by special attention to cooperation, re- 
sources, and stakeholder influence. Cooperation is especially im- 
portant for contextual analysis in relation to agency staff, informa- 
tion sources, and stakeholders, because of diverse influences on a 
program and the desirability of shared understandings. Resources to 
conduct contextual analysis include expertise, time, money, and in- 
kind contributions. It may take creative efforts to attract sufficient 
resources for the purpose or to limit evaluation goals to available 
resources. A coordinator of an evaluation can try to minimize dis- 
tortion of contextual analysis conclusions due to great differences 
in power among stakeholders. Most evaluations entail value judg- 
ments, which may lead to conflict when a contextual analysis 
reaches the conclusions and recommendations stage. Illustrative 
issues include parent organization priorities contrary to a preferred 
agency mission and cooperative versus competitive relationships 
with other providers in the service area. 

One role in evaluation coordination is reducing obstacles to con- 
textual analysis. For example, evaluation of management develop- 
ment programs is critically important because the connections be- 
tween education, performance, and benefits to the enterprise are 
indirect. Contexrual analysis can contribute to the rationale and 
justification for such programs. Among the obstacles are reluctance 
to evaluate, lack of resources, lack of know-how, low credibility, 
and difficulties using traditional research methods. The following 
coordination strategies are suggested for management development 
specialists to overcome these obstacles (Rothwell and Kazanas 
1993 ): 

• Overcome reluctance to evaluate by making a commitment to 
such evaluation and letting people know why and how it is 
done and the benefits related to the organization’s strategic 
plan and individual career development. 

• Overcome lack of resources by publicizing results, adapting 
methods based on available resources, finding out what stake- 
holders want to know to make decisions, and providing eval- 
uation as cost effectively as possible. 

• Overcome lack of know-how by increasing knowledge about 
evaluation in management development specialists and stake- 
holders, through reading about and visiting excellent manage- 
ment development programs to find out about their evaluation 
procedures. 

• Overcome low credibility by involving stakeholders in the 
evaluation process, including both producing and using 
conclusions. 
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• Overcome difficulties with traditional research methods by 
matching methods to stakeholder requirements. 

Assessing the desirability of program goals is sufficiently unusual 
that coordination of this aspect of evaluation is seldom viewed as 
an established part of someone’s role. Evaluation coordinators can 
help people decide how much effort to devote to assessing goals. 
Following are two examples regarding human resource develop- 
ment in enterprises that illustrate quite different levels and pace of 
assessment. 

An airline was on probation because of maintenance procedure 
violations. If more violations occurred in the next 6 months, their 
operating license might be suspended. They hired a consulting firm 
to study guidelines and procedures along with employee knowledge, 
skills, attitudes, and actual practices. There was evidence of unsat- 
isfactory supervisory monitoring and feedback, as well as low work- 
er morale and motivation. Tims, a supervisory training program 
was proposed and specific objectives for improved supervisory 
performance were agreed upon by supervisors and their managers. 
Evaluation coordination helped to identify goals. 

A contrasting coordination role was illustrated by a fastener manu- 
facturing company that was doing well with strong sales and secure 
budgets in various departments. The manager of human resource 
development noticed increasing mention of new types of fasteners. 
At a meeting of other department managers he discussed this, and 
they agreed that the company was doing well, but its product line 
was limited. No one was really up on emerging trends, and they 
decided to list likely trends for discussion at their next meeting. At 
that time they prepared a set of expertise goals for key personnel. 
They then asked the human resources development manager to 
identify learning opportunities on building trends and fastening 
systems relevant to their business. 

These two scenarios suggest quite different approaches to assessing 
goals. The airline confronted an urgent operational problem that 
threatened its existence, one that warranted coordination of an 
early comprehensive and accurate assessment of potential goals to 
ensure an effective educational program. By contrast, the manufac- 
turing company was functioning well, which warranted a systematic 
hut more modest and leisurely approach to exploring future direc- 
tions to maintain organizational health (Brinkerhoff 1987). Co- 
ordination of a program review entails consideration of involving 
stakeholders in a feasible process that often includes self-study, 
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external review, and sometimes meta evaluation Feasibility con- 
cerns for coordination of program evaluation include planning, 
manageable scale, simplicity, use of expertise, and timing. Both 
program relationships and utilization of findings are major issues in 
coordination of a program review. Someone should coordinate a 
self-study (such as accreditation), which can contribute to coopera- 
tion from stakeholders closest to the teaching/learning transac- 
tions. During self-study planning, coordination can include obtain- 
ing stakeholder understanding, suggestion, and commitment. 

During self-study data collection and analysis, coordination can 
include monitoring, assisting, pilot testing, encouraging, validating, 
and disseminating. Coordination of external reviews of a program 
can include liaison with a self-study, selection of external reviewers, 
and monitoring reviews and reporting. 

Even small-scale program evaluations can be enhanced if coordina- 
tors provide guidelines, identify' people to help, share examples, 
suggest peer review, discuss progress, and respond to requests. 
Especially throughout more complex program evaluations, encour- 
aging utilization can be enhanced by attention to appropriate re- 
porting methods for the recipients and feasible recommendations 
that fit other plans and activities. 

Central to a program review is assessment of program design and 
implementation related to the teaching/learning transaction. When 
conducting a program review, it is helpful for coordinators to have 
criteria to guide data collection and analysis. The following exam- 
ple illustrates such criteria for evaluating the quality of the educa- 
tional process for continuing education for the health professions 
that pertained to program design and implementation. Regarding 
instructional strategies, the following criteria were listed: 

• Sequence content to help learners meet learning objectives. 

• Ensure that instruction includes enough examples of concepts 
to be learned. 

• Provide opportunities for active involvement, especially it 
changing attitudes is an objective. 

• Provide opportunities for practice, especially if development or 
maintenance ot skills is an objective. 

• Give participants feedback on their progress. 

Regarding implementing programs, the following criteria were 
listed: 
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* Implement educational activities as designed. 

* Ensure that instruction occurs in a manner, time, and place 
convenient to participants. 

* Help participants obtain resources to meet educational 
objectives. 

Such criteria could help coordinators guide evaluation of program 
planning and implementation, including the teaching/learning 
transaction and the stakeholders most directly involved in the 
process (Green and Associates 1984). 

Additional examples of coordination pertain to use of quality cir- 
cles as part of a program review (Angelo and Cross 1993), assess- 
ment of educational materials (Duning, Van Kekerix, and 
Zabornwski 1993), evaluation of outcomes (Condelli and Kutner 
1997; Kirkpatrick 1994), and workplace literacy (Askov and 
Catalfamo 1997; Burkhart 1996; Carman 1995; and Jurmo 1995). 



Summary 

The following features are characteristic of effective coordination: 

Designate someone to coordinate large evaluation projects. 
Prepare a detailed and manageable plan. 

Obtain necessary resources and funding for the evaluation. 

Use a collaborative approach to obtain stakeholder cooperation. 
Deal with situational influences and reduce obstacles. 
Implement and monitor the evaluation project. 

Encourage ongoing evaluation. 



Sources : 



There are many potential sources of information for evaluation of 
educational programs for adults and manv designs that reflect cob 
lection and analysis of data from these sources. When deciding 
which sources to use in a specific instance, a major consideration 
is the evaluation issues that are important to stakeholders and the 
sources likely to he credible to them. Another consideration is 
provision for multiple indicators and perspectives that can contri- 
bute to the validity of the findings. Information could come from 
documents, individuals, and groups. A third consideration is the 
usefulness of a preliminary review of program characteristics that 
can identify important connections with the program aspect on 
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which an assessment is focused. Such characteristics include agen- 
cy goals, course objectives and methods, participant characteristics, 
evidence of program quality and results, and resources. Conclusions 
from various aspects of evaluation can contribute to this prelimi- 
nary review. 



The following two examples illustrate concepts and procedures 
regarding evaluation sources and designs for projects focused on 
contextual analysis and program review. A published literature 
review that critiqued many research reports on an aspect of con- 
tinuing education, in a form useful for contextual analysis, illu- 
strates a source of evaluation information. A hook by the American 
Medical Association (Davis and Fox 1994) is such an example. It 
was based on two North American conferences that assembled 
scholars related to continuing medical education to identify' broad 
topics, assemble teams to review scholarly writings on the topics, 
share their critiques, and report their critiques in a book on The 
Physician as L earner. 

The introduction provides an overview of forces in the external 
environment that affect continuing medical education: patient 
characteristics and expectations, new medical knowledge and 
technology, recertification, government, insurance systems, man- 
aged care, and varied providers of continuing medical education. 
This introduction provides a broad perspective for analysing the 
context of a specific continuing medical education program, and it 
suggests and clarifies variables likely to be pertinent. Chapter 5 on 
developmental perspectives on learning, and especially the section 
on life span theory, explains how career development and learning 
is shaped by a person’s environment related to family, occupation, 
and community. 

Chapter 1 1 on participation in continuing medical education notes 
the breadth of learning activities and the transactional nature of 
participation reflecting the participant, provider, and societal con- 
text that facilitate and deter initial and ongoing participation, per- 
sistence, and application. The proposed model includes external 
influences and the context of medical practice, such as patient 
characteristics, technological change, managed care, and continu- 
ous quality improvement. Conclusions from such publications can 
be used to design an evaluation proposal and interpret findings. 

Because program reviews include various components, the sources 
of assessment information are correspondingly broad. This scope is 
illustrated by typical self-studies and external reviews. Components 
might include goals, program offerings, context, course quality, 
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materials, effectiveness of teaching, course improvement, provision 
of resources, collaboration, and program benefits. A preliminary 
review of such components can guide specifications on which to 
focus the program review and thus the main sources of information 
to use. 

The following example of discrepancy analysis oi a literacy program 
illustrates one way to obtain information from multiple sources 
(Knox et al. 1974). The formative evaluation purpose and multiple 
role perspectives for program evaluation make discrepancy analysis 
a useful assessment approach, This typically entails using separate 
data collection instruments for each stakeholder role, hut combin- 
ing the data for analysis purposes. Most items obtain information 
about both intended (desired, should) practice and current (actual, 
is) practice, so that discrepancies can be recognized. For many pro- 
gram characteristics that are assessed, information can be collected 
from various stakeholders and sources (participants, instructors, 
observers, coordinators, director, and program records). 

For instance, a participant questionnaire might include items mi 
reasons for attending, actual benefits, supportive climate, and dif- 
ficulty level. Instructor questionnaire items might ask about desired 
and actual characteristics regarding responsibility for assessment 
and orientation of participants, topic emphases, methods of instruc- 
tion, and program goals. An observer’s rating form on course inter- 
action might rate teacher’s responsiveness, formality, and encour- 
agement; students’ attentiveness and active participation; and 
facilities and equipment. Program coordinators and the director 
might respond to questions similar to the instructor's, including 
how participant achievement is and should he evaluated. A pro- 
gram statistics form might he used to rate information from pro- 
gram records, such as numbers and characteristics of participants, 
instructors, facilities, courses, and finances. Such items yield hub 
quantitative and qualitative data. All stakeholders are asked for 
their suggestions for improvement. Displaying responses in a matrix 
(for comparisons related to role) can lead to identifying shared 
views and differences of opinion, each of which might be useful for 
strengthening the program. Discrepancies readily lead to recom- 
mendations for improvement (ibid.). 

Various evaluation designs and sources of information are reflected 
in the following examples related to needs assessment and out- 
comes. Potential participants are the prime source of useful data for 
needs assessment. An additional source is writings about adults as 
learners (Mcrriam and Caffarella 1991). The resulting generaliza- 
tions can help in designing the assessment, deciding what 



34 



3 J 



Application of Guidelines 



information to collect and how to do so, and interpreting findings 
from the actual needs assessment. Another source is other people 
associated with the potential participants who are knowledgeable 
about the proficiencies they want to acquire. These other associates 
and experts can provide additional perspectives that can enrich the 
resultant understanding of unmet educational needs. It is usually 
efficient and minimally disruptive to begin with accessible sources, 
and then supplement them to obtain sufficient evidence to achieve 
the purposes of a needs assessment. The use of information from 
multiple sources (such as potential participants, other people, 
reports) allows identification of common themes and distinctive 
perspectives. 

Evaluation of pro-gram impact tends to be of special interest to 
external stakeholders, so they are an important source of informa- 
tion, especially at the planning stage so that the assessment ad- 
dresses issues that they value. Because impact pertains to perform- 
ance and results, participants and people associated with them as 
they use what they learn rend to be major sources of evaluation 
data. Self-studies and follow-up studies typically draw from multiple 
sources. 

Use of multiple sources of evaluation data is illustrated by the fol- 
lowing example of an assessment of a pilot effort (Brinkerhoff 
1987). A furniture company had experienced declining sales and 
profits for some years, but prospects for an improved economy and 
new markets led to companywide productivity studies. As a result, 
a decision was made to capitalize on the coming strong market by 
instituting a team production approach. The director of human 
resource development agreed to provide team building and project 
management training, which an employee survey and lirerature 
review indicated were crucial tor a product team approach. 

A pilot program was begun with the first three product teams. 

After 1 year, a review of company profits and customer satisfaction 
was very positive, and a survey of team performance showed correct 
use of training content. A high-level review panel recommended 
extension of team training to all product teams, based on the fore- 
going impact evaluation and data that indicated that training costs 
were less than 3 percent of total production costs. 

Several additional leadings provide more details regarding designs 
and sources of evaluation data for contextual analyses. They focus 
on environmental scanning (Naishitt and Ahurdcne 1990; Simpson 
1989), performance assessment (Campbell 1995), professional 
education (Delaney 1997; Stake and Migotskv 1995), workplace 
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literacy (Mikulecky and Lloyd 1996), and sampling from extension 
populations (Verma and Burnett 1996). 



Summary 



Evaluation designs can obtain data from many sources. One is documents (organizational 
records, research reports). Another is individuals and groups (participants, instructors, ob- 
servers, administrators, and policy makers). It is desirable to include multiple indicators, 
especially those most likely to be credible to stakeholders. It is efficient to conduct a pre- 
liminary review and then select the most promising vanables. Categories of information 
include goals, offerings, context, quality, materials, achievement, teaching, resources, col- 
laboration, and benefits. A useful design is discrepancy analysis, which compares actual with 
desirable features. 



Data Collection . 



Most evaluation projects regarding educational programs for adults 
use a combination of data collection procedures. They often in* 
elude collection of both quantitative and qualitative data to reflect 
attention to both “how much” and “why.” The specific data collec- 
tion techniques can be distinguished L. whether they obtain data 
from documents, individuals, groups, or observation. Each method 
of obtaining evaluation information has strengths and limitations. 
One consideration when selecting the data collection methods to 
use in a specific instance is to minimize intrusion. Illustrative data 
collection procedures include observation checklists, performance 
ratings, tests, interviews, questionnaires, performance appraisal, 
document content analyses, case study, portfolio, performance 
records analyses, productivity measurement, performance audit, 
cost-benefit analysis, return on investment, focus groups, and 
external review. 

The following two examples illustrate concepts and procedures 
regarding collection of data for evaluation projects focused on spe- 
cification of goals and needs assessment. The process of data col- 
lection for contextual analysis is illustrated by an approach to en- 
vironmental scanning for continuing education that was used at 
the University of Georgia (Simpson 1989). The scanning and fu- 
tures forecasting procedures used to understand relevant emerging 
trends in the state sought to identify' early signals to guide program 
planning and organizational relations. Planning for scanning by the 
continuing education division included administrative commitment 
to costs and benefits, clarification of organizational mission and the 
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purpose ot scanning, and provision ot a coordinator whose rede in- 
cluded reinforcement ot staff commitment to the process. An early 
orientation workshop emphasized voluntary and open staff involve- 
ment as the participants discussed the mission of the division, the 
value ot environmental scanning, trial use of scanning procedures, 
and their willingness to volunteer to help with the scanning effort. 
The detailed design of the scanning process (beyond broad, volun- 
tary participation) included explanation of responsibilities and pro- 
cedures for use of a taxonomy to select and code content of publi- 
cations and media. The tour major taxonomy categories were 
social, technological, economic, and political (STEP) trends and 
events. Each scanner reported on specific media and prepared 
single-page abstracts that also assessed implications for the division. 
At least three times per year, interested staff members reviewed 
abstracts to identify' a manageable number of themes (about 6-10) 
and to discuss ramifications tor planning. The analysis methods 
included Delphi procedures, cross-impact analysis, and scenario 
development. In addition, there was documentation ot actions 
taken in the division as a result and feedback to staff to encourage 
use of conclusions and continued cooperation. Recently, the 
Internet has allowed rapid review of current information, but selec- 
tion, summary, and use of such data are continuing concerns. 

Data collection related to goal assessment is illustrated by the fol- 
lowing examples of Cooperative Extension annual plans of work 
and long-range plans. Such long-range organizational plans and 
individual plans ot work should include attention to goals, which 
can he assessed by various stakeholders regarding desirability and 
feasibility. This was illustrated by the Texas Agricultural Extension 
Service in their assessment of the quality of county plans of work, 
in the context of their assessment ot Extension’s long-range plan- 
ning for the ‘90s (Marshall 1990). In each county, staff, program 
council members, and other leading citizens participated in study 
groups to analyze critical issues of high priority for county extension 
programs, so that time and resources would he devoted to the is- 
sues of greatest concern. Statewide staff development programs 
were provided to help implement and evaluate the 1987*90 long- 
range issue-based Extension programs to help ensure that the pro- 
grams would be in the mainstream ot a changing society. This in- 
cluded help in preparing annual plans ot work to indicate goals, 
procedures, time, and resources to be devoted to address the se- 
lected issues. 

In December 1989 and January 1990, a Texas A & M professor 
extension program development specialist surveyed district exten- 
sion directors ;md countv extension agents to assess the quality ot 
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county annual plans of work. Questionnaires were returned from 
all 28 district directors and , 2 of a representative sample of 100 
county agents who received them, which was generalizahle to the 
650 agent positions in the state. In the 1 5 -item questionnaire for 
staff in each role, criteria related to goals and the quality of the 
plan of work included the following: goal and subject matter to be 
taught are closely related, situation is stated as a basis for goal and 
activities, and the program is relevant to a current critical issue in 
the county. Because people other than the specialist use annual 
plans for performance review, the response rate was high. The cri- 
tique of the plans of work was part of a decade-long effort to im- 
prove reporting in relation to planning and budgeting including 
sound comparisons from year to year, and the findings from the 
survey helped improve reports. 

The specialist had the main responsibility for planning and con- 
ducting the survey and reporting the results. The main resource 
was printing and mailing questionnaires, and especially more than a 
month of the specialist’s time over a 3 -month period. Extension 
staff at state, district, and county levels viewed assessment of re- 
porting as important, so cooperation was forthcoming. 

Additional data collection concepts and procedures are reflected in 
the following examples related to goals, participation assessment, 
and outcomes evaluation. The first example illustrates the use of 
focus groups for data collection and encouraging commitment to 
use of findings. They are especially applicable when potential par- 
ticipants arc at locations where small groups can readily meet for a 
session of an hour or two. Focus groups were used in a large enter- 
prise with staff in multiple locations. The enterprise director of 
training assessed almost two dozen focus groups in about 10 regions 
in which staff members were located. 

The total evaluation project obtained information from several 
sources, in addition to focus group members. External information 
included benchmarking with data from similar organizations and an 
opinion survey, which contributed to some preliminary goals or 
production tasks for individual roles and work groups. Because 
such explicit goals and global measures had been lacking, the focus 
groups provided an opportunity for staff members to comment on 
the preliminary goals, which could increase acceptance of the goals 
as part of performance review and staff development activities. 

Coordinating the use of focus groups was difficult and occurred 
during several months for planning and implementation, with the 
focus group meetings occurring during several weeks. A survey 
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could have taken less time and money, but the likely response rate 
and buy-in by staff members was low. Commitment of adequate 
resources was essential. The director served as facilitator of several 
focus "roups and coordinated the selection and supervision of other 
people who served as facilitators and recorders. The director work- 
ed with an external vendor and helped prepare guidelines for focus 
groups. The decision to use focus groups for evaluation should be 
based on the conclusion that it is an appropriate method to use and 
on a commitment of sufficient time and money to plan and con- 
duct the focus groups so that they are effective. 

The following example illustrates use of evaluation questions, fo- 
cused on goals, to enable stakeholders to assess desirability of goals 
related to participants, instructors, topics, and contexts. Evaluation 
of goals can be for purposes of planning, improvement, or account- 
ability, and questions about the soundness of goals and desired 
modification, can emphasize participants, instructors, topics, or 
contexts. Following are illustrative questions for each of these per- 
spectives and elements (Grotelueschen 1980). 

Participants Instructors Topics Contexts 

i Planning Have various How might Is a proposed topic Are proposed goals 

potential desirable relevant to consistent with the 

participants instructional goals proposed goals.* agency image.’ 

achieved their be attained.' 

expectations . 7 

Improvement How could program Would a revision ot Would other topics Does current 

goals better some instructor better contribute to knowledge of the 

correspond with aims better align achieving goals’ setting suggest 

learner objectives' them with related modification of the 

program amis’ goals? 

Justification Are the educational Are the Is this the best topic Were the attained 

goals appropriate instructional to achieve program goals educationally 

for these purposes w>mpatible goals . 1 important? 

participants . 1 with the program 

gl ><lb? 

Participation assessment questions can obtain information about 
participants’ expectations, achievement, and application. They can 
be answered from information derived from questionnaires, tests, 
interviews, focus groups, documents, observations, and portfolios. 

Much of assessment related to participants’ progress has focused on 
satisfaction and on work-related education. Rv contrast, the fol- 
lowing example is one of the few reported on evaluating liberal 
adult education. In a detailed research report by the Center for the 
Stud v of Liberal Education for Adults, Miller and McGuire (19611 
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documented the rationale, procedures, and instruments from a pro- 
ject of several years to evaluate various types of liberal and general 
education programs for adults. The emphasis was on continuing 
higher education, and a major purpose was to develop evaluation 
instruments (tests, questionnaires, simulations) that participants 
and instructors could use to assess achievement. The following 
example focuses on this purpose, which can enable participants to 
monitor their progress in mastering content (that can be amor- 
phous) and enable instructors to assess learner achievement and 
use conclusions to increase program responsiveness. 

The authors worked with almost 100 continuing higher education 
practitioners (engaged in liberal adult education) and evaluation 
experts in developing a rationale and drafting evaluation instru- 
ments (many of which were included in the report). An initial 
stage was to select widespread liberal adult education objectives to 
guide evaluation procedures. Included were development of new 
interests, increased knowledge of pertinent concepts, analysis of 
personal and social attitudes, appreciation of intellectual and 
aesthetic values, critical thinking, making informed judgments, 
understanding relationships, and empathy for divergent viewpoints. 

The project rationale acknowledged the lofty goals of liberal educa- 
tion but asserted that program objectives can be stated in more spe- 
cific terms than is usually the case, to specify the types of intellec- 
tual behavior that indicate that the program is achieving its pur- 
pose. In the development of evaluation instruments, three criteria 
were used: realism, interest, and flexibility. It was assumed that 
liberal adult education programs likely to help participants achieve 
the intended objectives would have features related t o enhanced 
proficiency. 

In the various content areas, the following progress was reported: 

• Defining the area regarding concepts, issues, information, and 
materials of special importance to adults 

• Identifying important objectives for actual programs in the area 

• Drafting evaluation instruments and suggesting additional types 
of instruments and procedures tor future development (em- 
phasis was given to case analysis) 

• Initiating small-scale field testing of some of the instruments 

The following highlights illustrate the development of evaluation 
instruments to collect data and their pertinence to assessment of 
participation. Participants complete an evaluation instrument part 
way through an educational program and receive feedback 
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regarding their responses and those of other participants, and they 
discuss the conclusions in relation to their current understanding 
several times during the program. The resulting feedback can be 
used by participants (and in summary' form by the instructor) to 
guide the developmental process of learning to make moral deci- 
sions. Case situations were developed as the basis for evaluation 
questions. The evaluation intent is to encourage participants to 
reflect on the fit between specific instances and general principles, 
and to reflect on the assumptions and implications of various 
courses of action that address material and ethical issues. Such 
feedback can be of educational value at various stages of a program 
(Miller and McGuire 1961). 

hollow-up studies are one way to assess extent of utilization. The 
following example illustrates use of a verified interview to assess 
administrative change (Hentschcl 1979), A 15-hour simulation 
workshop for administrators on evaluation procedures was assessed 
by verified personal follow-up interviews to find out the extent to 
which past participants actually used the evaluation techniques 
from the workshop simulation in their subsequent work. An inven- 
tor/ of nine specific evaluation activities included in the workshop 
content was covered in personal interviews in their offices before 
the workshop and 10 months afterwards. When administrators 
referred to use of evaluation techniques, the interviewer asked 
about examples, which administrators often shared for verification. 
Scoring gave weight to verified activities. Pro and post scores of 
workshop participants were compared with those of a comparison 
group of administrators who expressed interest in the workshop but 
who did not attend. There was no significant change in scores tor 
the comparison group, hut a significant improvement for the 
participants. 

Additional readings with more details on data collection include 
Salant and Dillman (1994), Krathwohl (1993b Gall, Borg, and 
Gall (1996), and Miles and Huherman (1984). Other examples 
pertain to English as a second language (Burt and Keenan 1995; 
Burt and Saccomano 1995), interactive television for health edu- 
cation (Byers et al. 19961, leadership development (Earnest 19%), 
family life (Fctsch 1997), self-help guides (Fleischman 1995), statt 
development (Neal 1997), literacy (Ntiri 1996), training (Phillips 
19961, and participation satisfaction (Popham 199'"' 
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Summary 



There are many potential sources of evaluation data: interview, focus group, performance 
audit, portfolio, questionnaire, document analysis, and observation checklist. It is desirable to 
use some combination of such sources to collect both quantitative and qualitative data. Spe- 
cific questions can address expectations, achievement, and application. Effective evaluation 
data collection minimizes intrusion and disruption of the program. 



Analysis. 



In practice, analysis of evaluation data varies greatly in a number 
of ways. One is formality. Sometimes data analysis is very informal, 
as when an instructor quickly reviews a set of participant tests or 
opinionnaires and draws some general impressions of learner 
achievement or satisfaction. Increasing levels of formality of data 
analysis reflect the importance of greater validity and specificity of 
conclusions. A second variation is the number of different perspec- 
tives on program expectations, quality, and effectiveness to be 
included in the data analysis. For example, data could be analyzed 
to compare assessments of the effectiveness of an instructional pro- 
cedure by multiple stakeholders such as instructor, participants, 
peers, and a coordinator. A third variation involves the types of 
data to include in one data analysis, such as judging staff perfor- 
mance based on both qualitative data (self-assessments, themes 
from open-ended comments hv participants), and quantitative data 
(scores from rating scales used in peer review, percentages based on 
participant achievement and persistence). A fourth variation is the 
number of points in time at which data were collected to allow 
time series data analysis to understand trends, such as in partici- 
pant progress or instructor improvement. A fifth is the use of infor- 
mation to interpret the findings, such as generalizations from a lit- 
erature review or findings from similar evaluation studies of com- 
parable programs. 

There are many texts that explain and illustrate data analysis pro- 
cedures. Worthen and Sanders (1987) and Wort hen, Sanders, and 
Fitzpatrick (1997) provide an overview. More specific procedures 
tor analysis of quantitative data are explained in Gall, Bore, and 
Gall (1996), Hopkins, Glass, and Hopkins (1967), and Krathwohl 
(1995). More specific procedures for analysis of qualitative data are 
explained in Cuba and Lincoln ( 1 °8 1 , 1989), Lincoln and Guha 
(1985), Miles and Huherman (1984), and Strauss and Corbin 
(1990). 
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Two examples illustrate concepts and procedures regarding data 
analysis focused on staffing and participation. Comparable infor- 
mation related to teaching in similar programs can contribute to 
analysis of staff performance, as illustrated by a report on com- 
parative data for evaluating extension teaching. In 1989, the Ohio 
State University established a system for evaluating extension 
teaching for purposes of improvement, performance appraisal, and 
promotion and tenure. Data collection instruments were developed 
for use by participants, peers, supervisors, and subject matter ex- 
perts (Spiegel 1992). 

A nine-item questionnaire was prepared to obtain feedback from 
groups attending Cooperative Extension programs. The items 
about the instructor pertained to the following: well-prepared, 
interested in helping, respect for all, stimulated learnings, clear 
answers, application of content, clear explanations, held attention, 
and helpful information. Participant responses were on a five-point 
agree/disagree scale. Participant responses were obtained for a vari- 
ety of extension staff, which created a database that contributed to 
interpretation of summaries for a staff member, by allowing com- 
parison with summary information for similar staff regarding role, 
content speciality, years of experience, and size of learning group. 
The instructor completed a form to accompany the participant 
questionnaires, which provided background information about the 
instructor and session, along with teaching methods. 

Other instruments were on one-to-one instruction and feedback 
from peers and experts. Extension instructor use of one or more of 
these instruments, and normative information to help interpret 
tabulations, illustrates an innovative use of evaluation tor staff 
review and development (Spiegel 1992). Comparative information 
from the database allowed analysis and interpretation of individual 
staff performance. 

Another approach to data analysis is illustrated by the following 
example of portfolio review as participation and empowerment 
evaluation (Miller and Associates 1993). One part of a large parent 
education project used a community development approach. Port- 
folio assessment was used as a supplementary wav of presenting the 
evaluation plan results. Local program portfolios and panel reviews 
of the portfolios were central to the assessment. Staff development 
was a major outcome. Portfolios contained descriptions, docu- 
ments, data, and statements arranged to best represent a project. 
Portfolios differed greatlv, hut fit their projects, by explaining the 
project and giving perspective to the evaluatin' results. 
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Preparation of portfolios was only half the evaluation review pro- 
cess. The other half was getting people to read and use the infor- 
mation. This was accomplished by organizing review panels of 
knowledgeable people who were diverse in backgrounds, view- 
points, specialties, ethnicity, and region. Panelists received 2 hours 
of orientation and devoted 4 days to the review. The orientation 
covered review tasks and timing, ideas about value and worth, 
statement of goals, and the general contents of portfolios. The re- 
view process was structured by each panel with panelists working 
both alone and together. A panelist completed a one-page sum- 
mary report on each portfolio, which included open questions on 
especially valuable features and on suggestions for the project. 
Although no interrater reliability was reported, similar projects had 
reported over 90 percent agreement by panel members. The port- 
folio review process revealed that a major outcome of the project 
was staff development, which is relevant to assessment related to 
participation (Miller and Associates 1993). Portfolio review is a 
form of analysis that can be used for various evaluation purposes. 

Additional data analysis concepts and procedures are reflected in 
the following examples related to needs assessment, program re- 
view, and materials assessment. Following are some ideas regarding 
data analysis for purposes of needs assessment. Analysis procedures 
vary with the type of data (quantitative or qualitative) and the pur- 
pose of the needs assessment (self-assessment or for marketing pur- 
poses). Sometimes, analysis consists of identifying from various 
types of data, common themes that help specif', needs and reduce 
the risk of program failure due to unresponsiveness. For quantita- 
tive data, typical procedures include frequency distributions, valid- 
ity checks, cross-tabulations, data reductions, and multivariate 
analysis. For qualitative data, typical procedures include inductive 
classifications, code themes, form working hypotheses, test con- 
cepts with specific passages, confirmation checks, and use of quotes 
to illustrate insights. It is desirable to me cross-validation across 
multiple sources to identify common themes, h is also important to 
recognize that interpretation combines facts and values. As stake- 
holders judge the soundness of conclusions it is helpful for them to 
use such criteria as relevance, usefulness, and precision. The needs 
assessment conclusions can also be compared with those from 
similar needs assessments. 

[•valuation of the overall program iMiallv includes perspectives ol 
various stakeholders and iiu lesion of various i vpes of data. I his 
typically occurs when a sell-sttidv is followed bv an external review. 
Doing so might inc hide both quantitative and qualitative data. 
Quantitative data analysts tvpa ally tiu hub's attention to sampling. 
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valid instruments, tabulation, data reduction, associations, multi- 
variate analysis, and significance. By contrast, qualitative data 
analysis can include analytic induction based on natural language 
narratives to identify themes, suggest explanations and test them 
with additional information, cross validation, and verification by 
subjects. Discussion of interpretation of the data analysis can 
include evaluation about achievement of evaluation objectives, 
ethical issues, value of conclusions, stakeholder review, and com- 
parison with similar evaluation reports. A usual result of a program 
review is some commendations of favorable features and recom- 
mendations about desired improvements. The details of the evalua- 
tion report provide a rationale and justification tor the recommen- 
dations. For many program reviews, involvement of stakeholders is 
essential for both the soundness of the conclusions and the likeli- 
hood of implementation of the recommendations (Patton 1986). 

Data analysis related to instructional materials can sometimes be 
closely connected with their use (Ferencc and Vockell 1994). For 
example, when introd’ cing new print materials or simulation activ- 
ities based on case n trials, such evaluative activities as peer re- 
view, participant critique of materials, and instructor appraisal of 
how well materials are working can serve both refinement of the 
materials and their broader assessment. This can be enhanced 
when the same materials are used by various instructors and their 
participants, if there is provision for sharing these assessments in a 
type of meta evaluation. The emergence of electronic methods 
such as computer-based education makes it feasible to accumulate 
and analyze data about use and achievement and to do so longitud- 
inally over time. The evaluation of materials can also focus on the 
relative effectiveness of alternative materials, regarding achieve- 
ment, motivation, and application. The essence of data analysis is 
conclusions relevant to the evaluation purpose and warranted by 
the data. 

More detailed sources of suggestions about data analysis are avail- 
able regarding specific program aspects. For needs assessment, re- 
views of pertinent publications can be especially useful (Houle 
1992; Kasworm 1983; Merriam and Caffarclla 1991; Pennington 
1980). For staffing assessment related to adult basic education, 
many examples are provided in Grotelucschen, Gooler, and Knox 
(1976) and in Knox and Associates (1974). Other examples in- 
clude management development (Brown-Wrighr 1996), employ- 
ment retraining (Kingct al. 1995), and portfolio assessment 
(Johnson and Bergman 1996). 
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Summary 

Evaluation data analysis may he formal or informal, hut usually reflects various viewpoints. 
To do so typically includes analysis ot various types of data. Quantitative data analysis pro- 
cedurcs include frequency distributions and cross-tabulations as well as multivariate analysis 
Qualitative data analysis procedures include inductive classifications, coding themes, and 
confirmation checks. Normative data such as a comparative database can contribute to in- 
terpretation of conclusions. Time scries analysis of longitudinal data is especially useful for 
understanding trends. Useful evaluation analysis results in commendations regarding favor- 
able features and recommendations regarding desirable improvements. 



Utilization 



A fundamental reason for conducting almost any evaluation is to 
have the conclusions used. Following arc general suggestions that 
apply to assessment of most program aspects. Stakeholders arc 
encouraged to use conclusions by their involvement in the process 
and by the credibility of the findings. The type and extent of in- 
volvement should be appropriate to the nature of the evaluation 
project and the backgrounds and interests of each category of 
stakeholder. Credibility depends on the purpose and design of the 
evaluation in relation to the issues it addressed and the resources 
devoted to the evaluation. Utilization of assessment conclusions 
and recommendations can be for purposes of planning, improve- 
ment, or accountability. Winning and maintaining stakeholder 
cooperation for evaluation is as important as mastery of evaluation 
procedures (Patton 1986). 

The following two examples illustrate concepts and procedures 
regarding use of conclusions, focused on participation and staffing. 
Utilization of evaluation conclusions related to participation is 
especially dependent on the assessment process, along with later 
efforts to encourage use. Use of findings to encourage participant 
enrollment and reflection also contribute to application. As partici- 
pants receive useful feedback about their achievement and pro- 
gress, they can increase their receptivity to subsequent assessment 
related to their participation and application of what they learn. 

An approach to assessment related to participation is illustrated by 
the following example on workplace literacy education tor union 
members. With funding from the U.S. Department ot Education’s 
National Workplace Literary' Program, a collaborative worker lit- 
eracy program for health care paraprofessionals was conducted by 
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City University ot New York and the New York City Central Labor 
Council (Perin 1992). Evaluation was related to several aspects of 
participation. Program applicants were screened for entry using a 
writing assessment used for assigning college students to remedial 
classes. The program was similar to college remedial courses, hut 
was contextualized regarding health care content. 

Literacy level was assessed in two ways: by simulated college places 
ment tests in reading, writing, and math and by teacher ratings. 
Both assessment methods showed gains for writing; the tests show- 
ed gains for math but not reading; and teacher ratings showed 
gains for reading hut not math. The follow-up survey in the third 
and fourth month after the program reported that 65 percent of a 
sample of 967 participants had been accepted to a college, mostly 
to study in health fields. Many passed college placement tests in 
reading, writing, and math. The labor unions were active partners 
in the program and continued their use of the instructional mate- 
rials beyond the funded project. The evaluation report included 
recommendations to select teachers committed to the program’s 
dual emphasis on health and literacy, increase teacher’s involve- 
ment in curriculum revision, and increase staff development on the 
dual emphases of health and literacy. This strategy used involve- 
ment by stakeholders to encourage their use of conclusions. 

Staffing assessment can contribute to organizational benefits, by 
encouraging organizational vitality, change, and teamwork. This is 
similar to the basic rationale for quality improvement with its em- 
phasis on structural influences, data-hnsed decisions, teamwork, 
and communication in which staff members have responsibility for 
both doing their job and learning how t o do it better. Action re- 
search, which involves continuing education staff members in eval- 
uation and utilization, provides an especially promising vehicle for 
both assessment and improvement. 

A way to encourage utilization is to make sure that assessment 
questions are pertinent. Questions to guide evaluation regarding 
staffing might include — 

• Why is the assessment being initiated? 

• What arc the sources and types of questions to guide the 
evaluation? 

• How will use of evaluation conclusions he encouraged? 

• What are the fail-back options in case there are difficulties with 
the plan? 



5 i 



47 



Application of Guidelines 



In addition to reporting and sharing evaluation conclusions to en- 
courage utilisation, assessment can include some aspects of meta 
evaluation, hy use of evaluation standards to critique and strength- 
en the assessment process. The findings from similar staff evalua- 
tion projects can he used to interpret findings (Grotelueschen 
1980; Sanders 1994). 

The following examples illustrate how to encourage utilization re- 
lated to various program aspects. Early understanding of guidelines 
for needs assessment and encouragement of stakeholder involve- 
ment can contribute much to use of conclusions. This is especially 
so for potential participants and practitioners who might use find- 
ings to plan and conduct effective and responsive programs. The 
following example of an assessment center and practice audit illu- 
strates an indepth form of needs assessment. Assessment centers 
and indepth practice audits are among the most complex and ex- 
pensive forms of educational needs assessment. Typically, they 
entail T3 days and a variety of assessment procedures to yield a 
comprehensive evaluation of needs and potential professional de- 
velopment. Phase l involves organization of a team to guide the 
assessment. Phase 2 is the identification of professional profici- 
encies and practice standards. Phase 3 is the construction of assess- 
ment materials, which includes both assessment techniques and 
content related to specific occupational practice standards. Phase 4 
entails use of these assessment materials in an assessment work- 
shop. Phase 5 includes analysis of the assessment results to identify 
individual needs hy comparing current proficiency (from assess- 
ment process) with desired proficiency (from practice standards). 
Phase 6 is provision of continuing education activities that are 
responsive to the identified needs. Phase 7 involves ongoing eval- 
uation of the programs regarding meeting needs and improving 
practice (Queeney 1995). The thoroughness and investment in an 
assessment center contribute to commitment to use conclusions. 

Because evaluation of situational influences can identify conflicting 
expectations, special attention should he given to externa! stake- 
holders. Attention to reporting and utilization is important 
throughout the assessment process. When identifying stakeholders, 
find out the issues that concern them and the forms of reporting 
most likely to promote use of conclusions. Evaluation standards can 
be used for meta evaluation to critique and improve the contextual 
analysis. Reporting should he responsive to stakeholders and audi- 
ences, avoid unnecessary complexity, and encourage utilization. 

Organizational expectations can he a major consideration when 
evaluating an executive education program. One wav to assess 
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program goals is to compare them with expectations o( the parent 
organization. Following are questions that can he asked about 
organizational expectations for an executive education program 
(Yicere 100C») . The conclusions could he i..*ed to modify goals 
and/or interpret goals to organisation members to improve the tit 
between program goals and stakeholder expectations. I liis would 
promote utilization. Answering such questions can contribute to 
utilization o( the total evaluation conclusions. 

• Does your organization recognize executive education as a 
competitive capability that can assist in the development and 
revitalization ot both individual leaders and the overall 
organization? 

• Do your organization’s executive education programs locus on 
building both the individual talents ot leaders and the collective 
knowledge base ot the organization? 

• Do your organization’s executive education progiams blend 
experience, training, education, and other torms ot challenge 
into <1 learning process? 

• Does your organization use the real-lime discussion ot reablite 
business issues as part ot your executive education process? 

• Does your organization recognize the power ot executive edu- 
cation as a force for individual and organizational development? 

I he foregoing example regarding organizational expectations also 
applies to evaluating goals and priorities. The emphasis in encour- 
aging use ot conclusions trom evaluation ot the desirability ot goals 
vanes with the intended use tor planning, improvement, and ac- 
cotintabilitv. Timing is also important so that conclusions are avail- 
able when stakeholders are rcadv m use them. 

Conclusions from evaluation ot goals, politics, and pnorities can he 
useful in various wavs. Included are decisions about program em- 
phasis, participant attraction, smiting, collaboration, community 
support, resource allocation, and organization. It the results ot miDi 
evaluations include anticipated benefits and desired future direc- 
tions valued bv stakeholders, and at least suggest implications re- 
garding hkelv costs and implementation, utilization related to the 
tollowing decisions can oct nr graduallv as stakeholders start to 
address action plans. 

One outcome ot evaluation related to participation is use ot Imd- 
mgs to conduct marketing eltorts tn attrau and retain participants. 
Following is an illustrative list ot such uses; 
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• Establish a marketing-relared research base (evaluation). 

• Write advertising copy chat emphasizes participant benefits 
from enrollments (including evaluation to increase 
effectiveness). 

• Integrate marketing concepts into daily routine (including 
assessing audience response). 

• Promote a comprehensive service orientation (including assess- 
ment of client satisfaction, ongoing evaluation, and identifi- 
cation of market niches). 

• Design marketing activities to enhance image of provider 
organization (including evaluating the fit between marketing 
activities and provider mission and goals). 

• Price programs and services competitively (including assessing 
client expectations and desirable fees in relation to alternative 
programs and cost recovery goals). 

• Develop an effective marketing mix of ways to communicate 
with potential participants (including ongoing assessment). 

• Obtain professional assistance with graphic design. (Use past 
evaluation to guide design and ongoing evaluation to assess its 
impact). 

• Track results. (The communication model of sending informa- 
tion and receiving a response that underlies marketing practice 
lends itself to such ongoing evaluation.) 

• Continuously analyze common marketing mistakes, to minimize 
them. (This is a special aspect of evaluation focused on 
improvement:.) 

Most of the ideas about marketing emphasize attracting partici- 
pants, but many of them apply also to retention. Program quality 
and responsiveness are especially important for retention, and eval- 
uation conclusions can be useful for this purpose (Cookson 1989; 
Simcrly 1989). 

The following example illustrates a way to increase utilization of 
evaluation conclusions by monitoring program implementation 
(Brinkerhoff 1987). The extent and type of evaluation should re- 
flect program characteristics and assessment purposes in various 
enterprise human resource development programs. For instance, 
monitoring a 3 -hour workshop for supervisors on a new perform- 
ance appraisal system might include the following evaluation 
activities: 



• Comparison of actual participants with those intended 

• Instructor notes during the session on suggestions for improve 
ments next time 

• A brief opinionnaire completed by participants 
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This minimal assessment seems warranted to: brief, well-develop- 

ed, frequently repeated workshop, because it encourages account" 
ability and responsiveness, notes whether it was implemented as 
planned, and captures ideas for revision. 

By contrast, when a new intensive 3 -day workshop for sales person- 
nel on the new product line was carefully planned for deliver; at 
branch locations throughout the world, an elaborate evaluation 
system was provided (ibid.). It included — 

• Records of preparation and attendees at each location 

• Participant reaction forms completed at the end of each day 

• A knowledge inventory completed at the end of the second 
day, with both a group summary, and scores given participants 
as feedback to guide their study during the final day. 

• A multiple"page feedback form completed by the coordinator of 
each session/location, covering documentation and suggestions 
for revision 

This elaborate assessment system was focused on quality control of 
a major program at decentralized locations. Conclusions allowed 
central office educators to monitor implementation, provide assis" 
tance where needed, and revise subsequent programs. 

When the need for a program arises suddenly, the evaluation may 
be more extensive and detailed to identify discrepancies between 
plan and implementation of an experimental program. For in- 
stance, in response to an upsurge in accidents, an experimental 
safety education program wa< provided for supervisors (ibid.). New 
safety procedures and materials to explain them were used and 
assessed in the following ways: 

• Participants completed a detailed end of session form, which 
was quickly summarized tor use in a pnstsession meeting of 
participants. 

• Participants met after each session to discuss reactions to the 
program, which were noted tor later analysis. 

• Two observers attended sessions and noted departures from the 
plan and participant interactions and reactions. 

Because of the experimental program, detailed information helped 
to identify discrepancies, which served as opportunities for the edu- 
cators to improve the program. These examples suggest the various 
strategies that may be required to encourage utilization. 
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Publications with additional suggestions regarding utilization ot 
conclusions include Patton (1986), Knox (1979a), Caffarella 
(1994), and Brinkerhoff (1987). Other examples focus cm coni' 
m unity leadership (Earnest et ah, 1995), management development 
(Ezzeddine and Holland 1996), family literacy (Taoct al. 1997), 
and use of radio (Wright et al. 1996). 



Summary 



Following are some wavs to encourage utilization: 



• Involve stakeholders appropriately in the evaluation process 

• Encourage ongoing cooperation 

• Provide a rationale for evaluation importance and procedures 

• Ensure credibility of conclusions by use ot pertinent evaluation questions and fit with the 
evaluation purpose and design 

• Recognize that evaluation conclusions can he used for program planning, improvement, 
and accountability 



Conclusion 



The foregoing review of evaluation concepts and procedures can he 
used in various ways by practitioners to strengthen educational pro- 
grams for adults. One way is to select one or more aspects of the 
total program on which ro focus an evaluation. A second is to re* 
view some broad action guidelines to use when planning and con- 
ducting a specific evaluation project. A third is ro identify some 
pertinent writings ro consult for greater detail on rationale and 
procedures. A fourth is to consider various examples that illustrate 
sets of interrelated decisions that constitute strategies tor 
evaluation. 
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